Reduced bandwidth stereo distortion correction for fisheye lenses of head-mounted displays

ABSTRACT

Systems and methods of providing stereo depth cameras for head-mounted display systems that require less memory and/or processing power. The stereo depth camera may include a left camera and a right camera spaced apart from each other by a distance. Each of the left and right cameras may be skewed outward by a non-zero angle from a forward direction of the head-mounted display system to provide a relatively wide field of view for the stereo depth camera. Each of the left and right cameras may include a camera sensor array and a camera lens positioned forward of the camera sensor array. Each of the camera lenses may include an optical axis that is laterally offset from the center of the associated camera sensor array toward a center of the support structure to center the left camera lens substantially on a scene center or principal point.

BACKGROUND Technical Field

The present disclosure generally relates to depth cameras forhead-mounted display systems.

Description of the Related Art

One current generation of virtual reality (“VR”) experiences is createdusing head-mounted displays (“HMDs”), which can be tethered to astationary computer (such as a personal computer (“PC”), laptop, or gameconsole), combined and/or integrated with a smart phone and/or itsassociated display, or self-contained. Generally, HMDs are displaydevices, worn on the head of a user, which have a small display devicein front of one (monocular HMD) or each eye (binocular HMD). The displayunits are typically miniaturized and may include CRT, LCD, Liquidcrystal on silicon (LCos), or OLED technologies, for example. Abinocular HMD has the potential to display a different image to eacheye. This capability is used to display stereoscopic images.

Demand for displays with heightened performance has increased with thedevelopment of smart phones, high-definition televisions, as well asother electronic devices. The growing popularity of virtual reality andaugmented reality systems, particularly those using HMDs, has furtherincreased such demand. Virtual reality systems typically envelop awearer's eyes completely and substitute a “virtual” reality for theactual or physical view (or actual reality) in front of the wearer,while augmented reality systems typically provide a semi-transparent ortransparent overlay of one or more screens in front of a wearer's eyessuch that actual view is augmented with additional information, andmediated reality systems may similarly present information to a viewerthat combines real-world elements with virtual elements. In many virtualreality and augmented reality systems, the movement of a wearer of sucha head-mounted display may be tracked in various manners, such as viasensors in the head-mounted display and/or external to it, in order toenable the images being shown to reflect user movements.

Positional tracking allows an HMD to estimate its position relative tothe environment around it, using a combination of hardware and softwareto detect absolute position. Positional tracking is an important featurein virtual reality, making it possible to track movement with sixdegrees of freedom (6DOF). Position tracking facilitates variousbenefits to the virtual reality experience. For example, positiontracking may change the viewpoint of a user to reflect different actionssuch as ducking, leaning forward, or jumping, and may allow for arepresentation of the user hands or other objects in the virtualenvironment. Position tracking also improves the 3D perception of thevirtual environment because of parallax (i.e., the way objects closer tothe eyes move faster than objects farther away).

There are different methods of positional tracking, including acoustictracking, inertial tracking, magnetic tracking, optical tracking, etc.,and/or combinations thereof. Inside-out tracking is a type positionaltracking that may be used to track the position of HMDs and/or relatedobjects (e.g., controllers). Inside-out tracking differs from outside-intracking by the location of the cameras or other sensors used todetermine the HMD's position. For inside-out tracking, the camera orsensors are located on the HMD, or object being tracked, while inoutside-out tracking the camera or sensors are placed in a stationarylocation in the environment.

An HMD that utilizes inside-out tracking utilizes one or more cameras to“look out” to determine how its position changes in relation to theenvironment. When the HMD moves, the sensors readjust their place in theroom and the virtual environment responds accordingly in real-time. Thistype of positional tracking can be achieved with or without markersplaced in the environment.

The cameras that are placed on the HMD observe features of thesurrounding environment. When using markers, the markers are designed tobe easily detected by the tracking system and placed in a specific area.With “markerless” inside-out tracking, the HMD system uses distinctivecharacteristics (e.g., natural features) that originally exist in theenvironment to determine position and orientation. The HMD system'salgorithms identify specific images or shapes and use them to calculatethe device's position in space. Data from accelerometers and gyroscopescan also be used to increase the precision of positional tracking.

BRIEF SUMMARY

A head-mounted display system may be summarized as including a supportstructure; and a stereo depth camera carried by the support structure,the stereo depth camera operative to capture stereo images, the stereodepth camera including a left camera skewed outward by a non-zero anglefrom a forward direction of the head-mounted display system, the leftcamera comprising a left camera sensor array and a left camera lenspositioned forward of the left camera sensor array, the left camera lenscomprising an optical axis that is laterally offset from the center ofthe left camera sensor array toward a center of the support structure tocenter the left camera lens substantially on a principal point; and aright camera horizontally spaced apart from the left camera and skewedoutward by a non-zero angle from the forward direction of thehead-mounted display system, the right camera comprising a right camerasensor array and a right camera lens positioned forward of the rightcamera sensor array, the right camera lens comprising an optical axisthat is laterally offset from the center of the right camera sensorarray toward the center of the support structure to center the rightcamera lens substantially on the principal point.

Each of the left camera lens and the right camera lens may include afisheye lens. The optical axis of the left camera lens may be laterallyoffset from the center of the left camera sensor array by a left offsetdistance and the optical axis of the right camera lens may be laterallyoffset from the center of the right camera sensor array by a rightoffset distance, wherein the left offset distance may be equal to theright offset distance. The left camera and right camera may each beskewed outward by a non-zero angle that is between 5 degrees and 10degrees from the forward direction. The respective lateral offsets ofthe left and right camera lenses may provide a horizontal disparitybetween corresponding points of images captured by the left and rightcameras that is less than 5 pixels. The left and right camera lenses maybe laterally offset such that the distortion of the camera lenses iscentered about a scene center that is in the forward direction of thehead-mounted display system.

A stereo depth camera operative to capture stereo images may besummarized as including a left camera skewed outward by a non-zero anglefrom a forward direction of a head-mounted display system, the leftcamera comprising a left camera sensor array and a left camera lenspositioned forward of the left camera sensor array, the left camera lenscomprising an optical axis that is laterally offset from the center ofthe left camera sensor array toward a center of the head-mounted displaysystem to center the left camera lens substantially on a principalpoint; and a right camera horizontally spaced apart from the left cameraand skewed outward by a non-zero angle from the forward direction of thehead-mounted display system, the right camera comprising a right camerasensor array and a right camera lens positioned forward of the rightcamera sensor array, the right camera lens comprising an optical axisthat is laterally offset from the center of the right camera sensorarray toward the center of the head-mounted display system to center theright camera lens substantially on the principal point.

Each of the left camera lens and the right camera lens may include afisheye lens. The optical axis of the left camera lens may be laterallyoffset from the center of the left camera sensor array by a left offsetdistance and the optical axis of the right camera lens may be laterallyoffset from the center of the right camera sensor array by a rightoffset distance, wherein the left offset distance may be equal to theright offset distance. The left camera and right camera may each beskewed outward by a non-zero angle that is between 5 degrees and 10degrees from the forward direction. The respective lateral offsets ofthe left and right camera lenses may provide a horizontal disparitybetween corresponding points of images captured by the left and rightcameras that is less than 5 pixels. The left and right camera lenses maybe laterally offset such that the distortion of the camera lenses iscentered about a scene center that is in the forward direction of thehead-mounted display system.

A method of providing a stereo depth camera operative to capture stereoimages may be summarized as including coupling a left camera to asupport structure of a head-mounted display system, the left cameraskewed outward by a non-zero angle from a forward direction of thehead-mounted display system, wherein the left camera comprises a leftcamera sensor array and a left camera lens positioned forward of theleft camera sensor array, the left camera lens comprising an opticalaxis that is laterally offset from the center of the left camera sensorarray toward a center of the support structure to center the left cameralens substantially on a principal point; and coupling a right camera tothe support structure of the head-mounted display system, the rightcamera skewed outward by a non-zero angle from the forward direction ofthe head-mounted display system, the right camera comprises a rightcamera sensor array and a right camera lens positioned forward of theright camera sensor array, the right camera lens comprising an opticalaxis that is laterally offset from the center of the right camera sensorarray toward the center of the support structure to center the rightcamera lens substantially on the principal point.

Coupling a left and right camera to a support structure of ahead-mounted display system may include coupling a left and right camerato a support structure of a head mounted display system, and each of theleft camera lens and the right camera lens may include a fisheye lens.Coupling a left and right camera to a support structure of ahead-mounted display system may include coupling a left and right camerato a support structure of a head mounted display system, and the opticalaxis of the left camera lens may be laterally offset from the center ofthe left camera sensor array by a left offset distance and the opticalaxis of the right camera lens is laterally offset from the center of theright camera sensor array by a right offset distance, wherein the leftoffset distance may be equal to the right offset distance. Coupling aleft and right camera to a support structure of a head-mounted displaysystem may include coupling a left and right camera to a supportstructure of a head mounted display system, and the left camera andright camera may each be skewed outward by a non-zero angle that isbetween 5 degrees and 10 degrees from the forward direction.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

In the drawings, identical reference numbers identify similar elementsor acts. The sizes and relative positions of elements in the drawingsare not necessarily drawn to scale. For example, the shapes of variouselements and angles are not necessarily drawn to scale, and some ofthese elements may be arbitrarily enlarged and positioned to improvedrawing legibility. Further, the particular shapes of the elements asdrawn, are not necessarily intended to convey any information regardingthe actual shape of the particular elements, and may have been solelyselected for ease of recognition in the drawings.

FIG. 1 illustrates a top plan view of a head-mounted display systemwhich includes binocular display, in particular manners in particularembodiments in accordance with the described techniques of the presentdisclosure.

FIG. 2 is a front pictorial diagram of a head-mounted display systemwhich includes binocular display subsystems and forward cameras that arecomponents of a stereo depth camera, in particular manners in particularembodiments in accordance with the described techniques of the presentdisclosure.

FIG. 3 is a top plan view of the head-mounted display system shown inFIG. 2, showing particular features of the cameras of the stereo depthcamera, in particular manners in particular embodiments in accordancewith the described techniques of the present disclosure.

FIG. 4A is a top plan view of a sensor and lens for a conventional rightside camera, with the lens centered over the sensor, in particularmanners in particular embodiments in accordance with the describedtechniques of the present disclosure.

FIG. 4B is a top plan view of a sensor and lens for a conventional leftside camera, with the lens centered over the sensor, in particularmanners in particular embodiments in accordance with the describedtechniques of the present disclosure.

FIG. 5A is a top plan view of a sensor and lens for a right side camera,with the lens laterally offset inward relative to the center of thesensor, in particular manners in particular embodiments in accordancewith the described techniques of the present disclosure.

FIG. 5B is a top plan view of a sensor and lens for a left side camera,with the lens laterally offset inward relative to the center of thesensor, in particular manners in particular embodiments in accordancewith the described techniques of the present disclosure.

FIG. 6 is a top plan view of respective sensors and lens assemblies ofthe two cameras shown in FIGS. 5A and 5B, in particular manners inparticular embodiments in accordance with the described techniques ofthe present disclosure.

FIG. 7 is a graph that includes a plot of percent distortion as afunction of field of view for lens assemblies and sensors of theconventional cameras shown in FIGS. 4A and 4B, in particular manners inparticular embodiments in accordance with the described techniques ofthe present disclosure.

FIG. 8 is a graph that includes a plot of percent distortion as afunction of field of view for lens assemblies and sensors of the camerasshown in FIGS. 5A and 5B, in particular manners in particularembodiments in accordance with the described techniques of the presentdisclosure.

FIG. 9 is a schematic block diagram for an example head-mounted displaysystem, in particular manners in particular embodiments in accordancewith the described techniques of the present disclosure.

DETAILED DESCRIPTION

In the following description, certain specific details are set forth inorder to provide a thorough understanding of various disclosedimplementations. However, one skilled in the relevant art will recognizethat implementations may be practiced without one or more of thesespecific details, or with other methods, components, materials, etc. Inother instances, well-known structures associated with computer systems,server computers, and/or communications networks have not been shown ordescribed in detail to avoid unnecessarily obscuring descriptions of theimplementations.

Unless the context requires otherwise, throughout the specification andclaims that follow, the word “comprising” is synonymous with“including,” and is inclusive or open-ended (i.e., does not excludeadditional, unrecited elements or method acts).

Reference throughout this specification to “one implementation” or “animplementation” means that a particular feature, structure orcharacteristic described in connection with the implementation isincluded in at least one implementation. Thus, the appearances of thephrases “in one implementation” or “in an implementation” in variousplaces throughout this specification are not necessarily all referringto the same implementation. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more implementations.

As used in this specification and the appended claims, the singularforms “a,” “an,” and “the” include plural referents unless the contextclearly dictates otherwise. It should also be noted that the term “or”is generally employed in its sense including “and/or” unless the contextclearly dictates otherwise.

The headings and Abstract of the Disclosure provided herein are forconvenience only and do not interpret the scope or meaning of theimplementations.

Systems and methods of the present disclosure are directed to providingstereo depth cameras to implement inside-out tracking for head-mounteddisplay systems that require reduced memory and/or processingrequirements. In at least some implementations, a stereo depth camera iscarried by a support structure of a head-mounted display system. Thestereo depth camera may include a left side camera and a right sidecamera, also referred to herein as left and right cameras or first andsecond cameras. The left and right cameras are spaced apart from eachother by a distance (e.g., 60-65 mm). In at least some implementations,each of the left and right cameras may be skewed horizontally outward bya non-zero angle (e.g., 5-10 degrees) from a forward direction of thehead-mounted display system to provide a relatively wide overall fieldof view (FOV) for the stereo depth camera. As discussed further below,the left camera may include a left camera sensor array and a left cameralens (or lens assembly), such as a fisheye lens, positioned forward ofthe left camera sensor array. The left camera lens may include anoptical axis that is laterally offset from the center of the left camerasensor array toward a center of the support structure to center the leftcamera lens substantially on a scene center or principal point.Similarly, the right camera may include a right camera sensor array anda right camera lens, such as a fisheye lens, positioned forward of theright camera sensor array. The right camera lens may include an opticalaxis that is laterally offset from the center of the right camera sensorarray toward the center of the support structure to center the rightcamera lens substantially on the scene center or principal point. Asdiscussed further below, these features align the pixels of imagescaptured by the left and right sensor arrays, which reduces oreliminates the need perform pixel shifts in memory, and allows forreduced memory requirements and/or allows available memory to be usedfor other purposes (e.g., distortion correction).

Generally, a depth sensing stereoscopic camera or “stereo depth camera”includes two sensors or cameras positioned a baseline distance (e.g.,about eye distance) apart from each other that are capable of sensingobject depth in a field of view. This may be accomplished via stereotriangulation or reconstruction, where depth data of pixels aredetermined from data acquired using a stereo or multiple-camera setupsystem. This way, it is possible to determine the depth to points in thescene, for example, from the center point of the line between theirfocal points. In order to solve the depth measurement problem using astereo camera system, it is necessary to first find corresponding pointsin the different images. Solving the correspondence problem is one ofthe main problems when using this type of technique. For instance,various types of noise, such as geometric noise from lens distortion orinterest point detection error, lead to inaccuracies in the measuredimage coordinates.

As an example, the disparity of features between two stereo images maybe computed as a shift to the left of an image feature in a left imagewhen viewed in a right image. For instance, a single point (or otherfeature) that appears at the x coordinate t (measured in pixels) in aleft image obtained by a left camera may be present at the x coordinatet−30 in a right image obtained by a right camera. In this case, thehorizontal disparity at that location in the right image would be 30pixels. The aligning of images from multiple cameras may undesirablyrequire significant memory or other resources (e.g., processing, datatransmission), which may be limited in various applications, such asreal-time applications and/or applications where it is desirable tominimize weight, size, or cost.

Further, for lenses having relatively high distortion at large fields ofview, such as fisheye lenses, large disparity of features between twostereo images may inhibit the identification of corresponding features.For example, since the lenses used may have a lot of distortion at largefields of view, a processor (e.g., image signal processor) may have adifficult time finding correlation between the left and right imagesbecause the distortion changes the shape of the feature(s) that thesystem is looking for as a function of field of view. Thus, if the useris looking at an area that is in the center of the FOV of one camera(e.g., left camera), where there is low distortion, in the center afeature has little or no distortion. If the same feature is positionedat the edge or periphery of the FOV of the other camera (e.g., rightcamera), the same feature has significant distortion. In such cases, thesystem may not determine that the points are matching due to the varyingdistortion of the feature in the two images.

Thus, there is a limitation on how much distortion can be allowed for inthe two cameras of a stereo depth camera. That limit may be driven byavailable memory. For example, if there is a particular amount of memoryavailable for the image signal processor, a small section of an imagecan be undistorted or corrected. For instance, an image signal processormay allow for only about 10 percent distortion, but a camera may have33-34 percent distortion at the periphery. As a result, withoututilizing the implementations discussed herein, depth may only becalculated over central portions of an image (e.g., portions that areless than 10 percent distortion in this example).

In at least some implementations of the present disclosure, rather thanshifting pixels in memory to align the images of the two cameras, foreach camera the lens is offset from the center of the sensor so that theoptical or alignment axis of the lens is not aligned to center of thesensor, but rather it is aligned to some offset value, which correspondsto the amount that the lens would be offset due to the outward tilt ofthe camera. Accordingly, the pixels in images of the left and rightcameras are at least substantially aligned with each other withouthaving to do the shift in memory. By offsetting the lens to be centeredon the scene center rather than the sensor center, the system does notneed to offset the images by a number of pixels (e.g., 30 pixels) inmemory to align the two cameras on the scene. Various features of thepresent disclosure are discussed further below with reference to theFigures.

FIG. 1 is a simplified top plan view of an HMD system 100 that includesa pair of near-to-eye display systems 102 and 104. The near-to-eyedisplay systems 102 and 104 include displays 106 and 108, respectively(e.g., OLED micro-displays), and respective optical lens systems 110 and112 that each have one or more optical lenses. The display systems 102and 104 may be mounted to a support structure or frame 114 or othermounting structure which includes a front portion 116, a left temple 118and right temple 120. The two display systems 102 and 104 may be securedto the frame 114 in an eye glasses arrangement which can be worn on thehead 122 of a user 124. The left temple 118 and right temple 120 mayrest over the user's ears 126 and 128, respectively, while a noseassembly (not shown) may rest over the user's nose 130. The frame 114may be shaped and sized to position each of the two optical systems 110and 112 in front of one of the user's eyes 132 and 134, respectively.Although the frame 114 is shown in a simplified manner similar toeyeglasses for explanatory purposes, it should be appreciated that inpractice more sophisticated structures (e.g., goggles, integratedheadband, helmet, straps, etc.) may be used to support and position thedisplays systems 102 and 104 on the head 122 of user 124.

The HMD system 100 of FIG. 1 is capable of presenting a virtual realitydisplay to the user 124, such as via corresponding video presented at adisplay rate such as 30 frames (or images) per second or 90 frames persecond, while other embodiments of a similar system may present anaugmented reality display to the user 124. Each of the displays 106 and108 may generate light which is transmitted through and focused by therespective optical systems 110 and 112 onto the eyes 132 and 134,respectively, of the user 124. While not illustrated here, each of theeyes includes a pupil aperture through which light passes into the eye,with a typical pupil size ranging from 2 mm (millimeters) in diameter invery bright conditions to as much as 8 mm in dark conditions, while thelarger iris in which the pupil is contained may have a size ofapproximately 12 mm—the pupil (and enclosing iris) may typically movewithin the visible portion of the eye under open eyelids by severalmillimeters in the horizontal and/or vertical directions, which willalso move the pupil to different depths from the optical lens or otherphysical elements of the display for different horizontal and verticalpositions as the eyeball swivels around its center (resulting in a threedimensional volume in which the pupil can move). The light entering theuser's pupils is seen by the user 124 as images and/or video. In someimplementations, the distance between each of the optical systems 110and 112 and the user's eyes 132 and 134 may be relatively short (e.g.,less than 30 mm, less than 20 mm), which advantageously causes the HMDsystem 100 to appear lighter to the user since the weight of the opticalsystems and the display systems are relatively close to the user's face,and also may provide the user with a greater field of view.

The HMD system 100 may also include forward cameras 136 a and 136 bwhich may be cameras of a stereo depth camera 136. The stereo depthcamera 136 may be operative to capture image data that may beselectively presented to the user 124, for example, in augmented realityapplications or in conjunction with virtual reality applications.Additionally or alternatively, the stereo depth camera 136 may be usedby a position tracking system of the HMD system 100 to track theposition of the HMD system 100 during use, as discussed elsewhereherein. As an example, each of the cameras 136 a and 136 b may comprisea video camera and associated lens system that captures images at aframe rate (e.g., 30 Hz, 60 Hz, 90 Hz) in a front camera field of viewthat has a relatively wide angle (e.g., 60°, 90°, 120°, 150°).

While not illustrated in FIG. 1, some embodiments of such an HMD systemmay include various additional internal and/or external sensors, such asto perform pupil tracking separately for each eye 132 and 134, to trackhead location and orientation (e.g., as part of head tracking), to trackvarious other types of movements and position of the user's body, othercameras to record external images (e.g., of an environment), etc.

Further, while the described techniques may be used in some embodimentswith a display system similar to that illustrated in FIG. 1, in otherembodiments other types of display systems may be used, including with asingle optical lens and display device, or with multiple such opticallenses and display devices. Non-exclusive examples of other such devicesinclude cameras, telescopes, microscopes, binoculars, spotting scopes,surveying scopes, etc. In addition, the described techniques may be usedwith a wide variety of display panels or other display devices that emitlight to form images, which one or more users view through one or moreoptical lens. In other embodiments, the user may view one or more imagesthrough one or more optical lens that are produced in manners other thanvia a display panel, such as on a surface that reflects light fromanother light source in part or in whole.

FIG. 2 shows a front view of an example HMD system 200 when worn on thehead of a user 202. FIG. 3 shows a top plan view of the HMD system 200,showing example fields of view 208 a and 208 b for forward cameras 206 aand 206 b, respectively, of the HMD system 200. The HMD system 200includes a support structure 204 that supports the front facing orforward stereo depth cameras 206 a and 206 b. The camera 206 a may bereferred to herein as the left camera 206 a and the camera 206 b may bereferred to herein as the right camera 206 b. The stereo depth cameras206 a and 206 b may be similar or identical to the cameras 136 a and 136b discussed above with reference to FIG. 1.

As shown in FIG. 3, the cameras 206 a and 206 b are directed forwardtoward a scene or environment 214 in which the user 202 operates the HMDsystem 200. The environment 214 may include one or more objects 213 (oneshown) therein, which may include walls, ceilings, furniture, stairs,cars, trees, tracking markers, or any other types of objects.

The cameras 206 a and 206 b may have respective fields of view 208 a and208 b. As a non-limiting example, the fields of view 208 a and 208 b maybe relatively large angle (e.g., 60°, 90°, 120°, 150°). As indicated bythe arrow 210 a, the left camera 206 a may be skewed or tiltedhorizontally outward by a non-zero angle 212 a from a forward direction(indicated by arrow 216 a) of the head-mounted display system 200.Similarly, as indicated by the arrow 210 b, the right camera 206 b maybe skewed or tilted horizontally outward by a non-zero angle 212 b froma forward direction (indicated by arrow 216 b) of the head-mounteddisplay system 200. For example, the non-zero angles 212 a and 212 b maybe between 5 and 10 degrees (e.g., 5 degrees, 7, degrees, 10 degrees).The two cameras 206 a and 206 b each have different pointing angles(“toed-out”) to capture images over a relatively large field of view(e.g., 150° to 180°), compared to implementations where the cameras arepointed directly forward or inward (“toed-in”).

FIGS. 4A and 4B illustrate a front view of a sensor array 302 a and lens204 a for a left camera 300 a (FIG. 4B), and a sensor array 302 b and alens 304 b for a right camera 300 b (FIG. 4A). The horizontal centers ofthe sensor arrays 302 a and 302 b are indicated by the dashed lines 306a and 306 b, respectively, and the vertical centers of the sensor arraysare indicated by the dashed lines 308 a and 308 b, respectively. In thisexample that shows a conventional configuration, the lens 304 a ispositioned directly over the center of the sensor array 302 a, and thelens 304 b is positioned directly over the center of the sensor array302 b. Since the two cameras 300 a and 300 b are skewed outward, asdiscussed above and shown in FIG. 3, a scene center or principal point310 for both of the cameras 300 a and 300 b is offset inward toward eachother by a certain amount, as indicated in FIGS. 4A and 4B. Inparticular, for the left camera 300 a, the center 310 is positionedinward (to the left as shown) of the horizontal center 306 a of thesensor array 302 a. Similarly, for the right camera 300 b, the center310 is positioned inward (to the right as shown) of the horizontalcenter 306 b of the sensory array 302 b. In conventional systems, inorder to center the cameras 300 a and 300 b, the software of thehead-mounted display system finds the center points 310 and utilizes asubstantial amount memory to shift the image over to correct thedistortion over the FOV. The centering undesirably costs memory thatcould otherwise be used to undistort the image. That is, the softwareshifts (or translates) pixels over to determine a new center, to use fordistortion correction.

As an example, an image signal processor of a head-mounted displaysystem may be able to store 60 or 70 pixels (columns of pixels), so animage may be offset by up to that number of pixels. Due to the camerasbeing tilted outward, approximately 30 pixels may be stored during readout before the actual data is used for correlation, because the pixelsthe images start correlating after 30 pixels have been read out becausethat is where the center is between the two cameras.

FIGS. 5A, 5B, and 6 illustrate an exemplary implementation of the leftcamera 300 a (FIGS. 5B and 6) and the right camera 300 b (FIGS. 5A and6). In this implementation, as shown in FIGS. 5B and 6, the left cameralens 304 a and its corresponding optical axis 312 a are laterally offsetby a distance “h” from the horizontal center 306 a of the left camerasensor array 302 a toward a center of the head-mounted display system tocenter the left camera lens substantially on the scene center orprincipal point 310. Similarly, as shown in FIGS. 5A and 6, the rightcamera lens 304 b and its corresponding optical axis 312 b are laterallyoffset by a distance “h” from the horizontal center 306 b of the leftcamera sensor array 302 b toward the center of the head-mounted displaysystem to center the right camera lens substantially on the scene centeror principal point 310. In at least some implementations, the lateraloffset distance “h” may be the same for the left camera lens 304 a andthe right camera lens 304 b.

Thus, rather than using memory for pixel offset, as with the cameraconfiguration shown in FIGS. 4A and 4B, by offsetting the lenses 304 aand 304 b inwardly with respect to the sensory arrays 302 a and 302 b,respectively, distortion is now centered about the scene center 310.Accordingly, correlation windows for images produced by the cameras 300a and 300 b match well because both images are centered on the scenecenter 310.

FIG. 7 is a graph 320 that includes a plot 322 of percent distortion asa function of field of view for conventional lens assemblies and sensorsshown in FIGS. 4A and 4B. As shown, the distortion is offset withrespect to the field of view due to the outward tilt of the cameras 300a and 300 b.

FIG. 8 is a graph 330 that includes a plot 332 of percent distortion asa function of field of view for lens assemblies and sensors shown inFIGS. 5A, 5B, and 6. As shown, the distortion is centered about thescene center (i.e., minimal distortion at the center of the FOV) due tothe lenses 304 a and 304 b being laterally offset inward with respect tothe centers of the sensor arrays 302 a and 302 b, respectively, asdiscussed above.

FIG. 9 shows a schematic block diagram of an HMD system 400 according toone or more implementations of the present disclosure. The HMD system400 may be similar or identical to the HMD systems 100 and 200 discussedabove. Thus, the discussion above with regard to the HMD systems 100 and200 may also apply to the HMD system 400.

The HMD system 400 includes a processor 402, a first camera 404 (e.g.,left camera) and a second camera 406 (e.g., right camera), which camerasare components of a stereo depth camera. The first camera 404 mayinclude a sensor array 404 b and a lens 404 a laterally offset from acenter of the sensor array, as discussed above with reference to FIGS.5A-5B and 6. The second camera 406 may include a sensor array 406 b anda lens 406 a laterally offset from a center of the sensor array, asdiscussed above.

The HMD system 400 may include a display subsystem 408 (e.g., twodisplays and corresponding optical systems). The HMD system 400 may alsoinclude a nontransitory data storage 410 that may store instructions ordata for distortion correction 412, position tracking, instructions ordata for display functionality 414 (e.g., games), and/or other programs416.

The HMD system 400 may also include various I/O components 418, whichmay include one or more user interfaces (e.g., buttons, touch pads,speakers), one or more wired or wireless communications interfaces, etc.As an example, the I/O components 418 may include a communicationsinterface that allows the HMD system 400 to communicate with an externaldevice 420 over a wired or wireless communications link 422. Asnon-limiting examples, the external device 420 may include a hostcomputer, a server, a mobile device (e.g., smartphone, wearablecomputer), etc. The various components of the HMD system 400 may behoused in a single housing (e.g., support structure 204 of FIGS. 2 and3), may be housed in a separate housing (e.g., host computer), or anycombinations thereof.

It will be appreciated that the illustrated computing systems anddevices are merely illustrative and are not intended to limit the scopeof the present disclosure. For example, HMD 400 and/or external devices420 may be connected to other devices that are not illustrated,including through one or more networks such as the Internet or via theWeb. More generally, such a computing system or device may comprise anycombination of hardware that can interact and perform the describedtypes of functionality, such as when programmed or otherwise configuredwith appropriate software, including without limitation desktopcomputers, laptop computers, slate computers, tablet computers or othercomputers, smart phone computing devices and other cell phones, Internetappliances, PDAs and other electronic organizers, database servers,network storage devices and other network devices, wireless phones,pagers, television-based systems (e.g., using set-top boxes and/orpersonal/digital video recorders and/or game consoles and/or mediaservers), and various other consumer products that include appropriateinter-communication capabilities. For example, the illustrated systems400 and 420 may include executable software instructions and/or datastructures in at least some embodiments, which when loaded on and/orexecuted by particular computing systems or devices, may be used toprogram or otherwise configure those systems or devices, such as toconfigure processors of those systems or devices. Alternatively, inother embodiments, some or all of the software systems may execute inmemory on another device and communicate with the illustrated computingsystem/device via inter-computer communication. In addition, whilevarious items are illustrated as being stored in memory or on storage atvarious times (e.g., while being used), these items or portions of themcan be transferred between memory and storage and/or between storagedevices (e.g., at different locations) for purposes of memory managementand/or data integrity.

Thus, in at least some embodiments, the illustrated systems aresoftware-based systems including software instructions that, whenexecuted by the processor(s) and/or other processor means, program theprocessor(s) to automatically perform the described operations for thatsystem. Furthermore, in some embodiments, some or all of the systems maybe implemented or provided in other manners, such as at least partiallyin firmware and/or hardware means, including, but not limited to, one ormore application-specific integrated circuits (ASICs), standardintegrated circuits, controllers (e.g., by executing appropriateinstructions, and including microcontrollers and/or embeddedcontrollers), field-programmable gate arrays (FPGAs), complexprogrammable logic devices (CPLDs), etc. Some or all of the systems ordata structures may also be stored (e.g., as software instructionscontents or structured data contents) on a non-transitorycomputer-readable storage medium, such as a hard disk or flash drive orother non-volatile storage device, volatile or non-volatile memory(e.g., RAM), a network storage device, or a portable media article(e.g., a DVD disk, a CD disk, an optical disk, a flash memory device,etc.) to be read by an appropriate drive or via an appropriateconnection. The systems, modules and data structures may also in someembodiments be transmitted as generated data signals (e.g., as part of acarrier wave or other analog or digital propagated signal) on a varietyof computer-readable transmission mediums, including wireless-based andwired/cable-based mediums, and can take a variety of forms (e.g., aspart of a single or multiplexed analog signal, or as multiple discretedigital packets or frames). Such computer program products may also takeother forms in other embodiments. Accordingly, the present disclosuremay be practiced with other computer system configurations.

Those of skill in the art will recognize that many of the methods oralgorithms set out herein may employ additional acts, may omit someacts, and/or may execute acts in a different order than specified.

The various implementations described above can be combined to providefurther implementations. These and other changes can be made to theimplementations in light of the above-detailed description. In general,in the following claims, the terms used should not be construed to limitthe claims to the specific implementations disclosed in thespecification and the claims, but should be construed to include allpossible implementations along with the full scope of equivalents towhich such claims are entitled. Accordingly, the claims are not limitedby the disclosure.

The invention claimed is:
 1. A head-mounted display system, comprising:a support structure; and a stereo depth camera carried by the supportstructure, the stereo depth camera operative to capture stereo images,the stereo depth camera comprising: a left camera skewed outward by anon-zero angle from a forward direction of the head-mounted displaysystem, the left camera comprising a left camera sensor array and a leftcamera lens positioned forward of the left camera sensor array, the leftcamera lens comprising an optical axis that is laterally offset from thecenter of the left camera sensor array toward a center of the supportstructure to center the left camera lens substantially on a principalpoint; and a right camera horizontally spaced apart from the left cameraand skewed outward by a non-zero angle from the forward direction of thehead-mounted display system, the right camera comprising a right camerasensor array and a right camera lens positioned forward of the rightcamera sensor array, the right camera lens comprising an optical axisthat is laterally offset from the center of the right camera sensorarray toward the center of the support structure to center the rightcamera lens substantially on the principal point.
 2. The head-mounteddisplay system of claim 1 wherein each of the left camera lens and theright camera lens comprises a fisheye lens.
 3. The head-mounted displaysystem of claim 1 wherein the optical axis of the left camera lens islaterally offset from the center of the left camera sensor array by aleft offset distance and the optical axis of the right camera lens islaterally offset from the center of the right camera sensor array by aright offset distance, wherein the left offset distance is equal to theright offset distance.
 4. The head-mounted display system of claim 1wherein the left camera and right camera are each skewed outward by anon-zero angle that is between 5 degrees and 10 degrees from the forwarddirection.
 5. The head-mounted display system of claim 1 wherein therespective lateral offsets of the left and right camera lenses provide ahorizontal disparity between corresponding points of images captured bythe left and right cameras that is less than 5 pixels.
 6. Thehead-mounted display system of claim 1 wherein the left and right cameralenses are laterally offset such that the distortion of the cameralenses is centered about a scene center that is in the forward directionof the head-mounted display system.
 7. A stereo depth camera operativeto capture stereo images, the stereo depth camera comprising: a leftcamera skewed outward by a non-zero angle from a forward direction of ahead-mounted display system, the left camera comprising a left camerasensor array and a left camera lens positioned forward of the leftcamera sensor array, the left camera lens comprising an optical axisthat is laterally offset from the center of the left camera sensor arraytoward a center of the head-mounted display system to center the leftcamera lens substantially on a principal point; and a right camerahorizontally spaced apart from the left camera and skewed outward by anon-zero angle from the forward direction of the head-mounted displaysystem, the right camera comprising a right camera sensor array and aright camera lens positioned forward of the right camera sensor array,the right camera lens comprising an optical axis that is laterallyoffset from the center of the right camera sensor array toward thecenter of the head-mounted display system to center the right cameralens substantially on the principal point.
 8. The stereo depth camera ofclaim 7 wherein each of the left camera lens and the right camera lenscomprises a fisheye lens.
 9. The stereo depth camera of claim 7 whereinthe optical axis of the left camera lens is laterally offset from thecenter of the left camera sensor array by a left offset distance and theoptical axis of the right camera lens is laterally offset from thecenter of the right camera sensor array by a right offset distance,wherein the left offset distance is equal to the right offset distance.10. The stereo depth camera of claim 7 wherein the left camera and rightcamera are each skewed outward by a non-zero angle that is between 5degrees and 10 degrees from the forward direction.
 11. The stereo depthcamera of claim 7 wherein the respective lateral offsets of the left andright camera lenses provide a horizontal disparity between correspondingpoints of images captured by the left and right cameras that is lessthan 5 pixels.
 12. The stereo depth camera of claim 7 wherein the leftand right camera lenses are laterally offset such that the distortion ofthe camera lenses is centered about a scene center that is in theforward direction of the head-mounted display system.
 13. A method ofproviding a stereo depth camera operative to capture stereo images, themethod comprising: coupling a left camera to a support structure of ahead-mounted display system, the left camera skewed outward by anon-zero angle from a forward direction of the head-mounted displaysystem, wherein the left camera comprises a left camera sensor array anda left camera lens positioned forward of the left camera sensor array,the left camera lens comprising an optical axis that is laterally offsetfrom the center of the left camera sensor array toward a center of thesupport structure to center the left camera lens substantially on aprincipal point; and coupling a right camera to the support structure ofthe head-mounted display system, the right camera skewed outward by anon-zero angle from the forward direction of the head-mounted displaysystem, the right camera comprises a right camera sensor array and aright camera lens positioned forward of the right camera sensor array,the right camera lens comprising an optical axis that is laterallyoffset from the center of the right camera sensor array toward thecenter of the support structure to center the right camera lenssubstantially on the principal point.
 14. The method of claim 13 whereincoupling a left and right camera to a support structure of ahead-mounted display system comprises coupling a left and right camerato a support structure of a head mounted display system, and each of theleft camera lens and the right camera lens comprises a fisheye lens. 15.The method of claim 13 wherein coupling a left and right camera to asupport structure of a head-mounted display system comprises coupling aleft and right camera to a support structure of a head mounted displaysystem, and the optical axis of the left camera lens is laterally offsetfrom the center of the left camera sensor array by a left offsetdistance and the optical axis of the right camera lens is laterallyoffset from the center of the right camera sensor array by a rightoffset distance, wherein the left offset distance is equal to the rightoffset distance.
 16. The method of claim 13 wherein coupling a left andright camera to a support structure of a head-mounted display systemcomprises coupling a left and right camera to a support structure of ahead mounted display system, and the left camera and right camera areeach skewed outward by a non-zero angle that is between 5 degrees and 10degrees from the forward direction.